Heterogeneous Face Recognition (HFR) is a challenging issue because of the large domain discrepancy and a lack of heterogeneous data. This paper considers HFR as a dual generation problem, and proposes a novel Dual Variational Generation (DVG) framework. It generates large-scale new paired heterogeneous images with the same identity from noise, for the sake of reducing the domain gap of HFR. Specifically, we first introduce a dual variational autoencoder to represent a joint distribution of paired heterogeneous images. Then, in order to ensure the identity consistency of the generated paired heterogeneous images, we impose a distribution alignment in the latent space and a pairwise identity preserving in the image space. Moreover, the HFR network reduces the domain discrepancy by constraining the pairwise feature distances between the generated paired heterogeneous images. Extensive experiments on four HFR databases show that our method can significantly improve state-of-the-art results. When using the generated paired images for training, our method gains more than 18\% True Positive Rate improvements over the baseline model when False Positive Rate is at $10^{-5}$.

UPDATE: After reading other reviewers' comments and the rebuttal, I decided to raise my score by one point from 6 to 7. I am satisfied with the effort the authors made to address my two major concerns and I recommend to accept this submission in agreement with the other reviewers. Overview/Contribution: The paper proposes a dual variational autoencoder to generate synthetic training data to combat the limited data in heterogeneous face recognition. The synthetic data tries to preserve identity via identity preserving generation both in the image and embedding spaces while providing sufficient variation for the training data of the downstream recognition task. Strengths: - Most facial recognition tasks involve certain assumptions that constrain the task into homogeneous set of inputs. Heterogeneous face recognition (HFR) is an important task for many practical applications that is attracting attention recently.

The paper proposes a dual variational autoencoder, used to generate new synthetic training data for heterogeneous face recognition, by preserving generation both in the image and embedding spaces and providing variation for the training data of the downstream recognition task. The authors claim a big improvement in performance. Reviewers initially were convinced on the goodness of the paper and then after the rebuttal one of the reviewer increased its rate. Thus the consensuswas reached and also the area chair agreed with the acceptance rate.

Heterogeneous Face Recognition (HFR) is a challenging issue because of the large domain discrepancy and a lack of heterogeneous data. This paper considers HFR as a dual generation problem, and proposes a novel Dual Variational Generation (DVG) framework. It generates large-scale new paired heterogeneous images with the same identity from noise, for the sake of reducing the domain gap of HFR. Specifically, we first introduce a dual variational autoencoder to represent a joint distribution of paired heterogeneous images. Then, in order to ensure the identity consistency of the generated paired heterogeneous images, we impose a distribution alignment in the latent space and a pairwise identity preserving in the image space. Moreover, the HFR network reduces the domain discrepancy by constraining the pairwise feature distances between the generated paired heterogeneous images.

Heterogeneous Face Recognition (HFR) is a challenging issue because of the large domain discrepancy and a lack of heterogeneous data. This paper considers HFR as a dual generation problem, and proposes a novel Dual Variational Generation (DVG) framework. It generates large-scale new paired heterogeneous images with the same identity from noise, for the sake of reducing the domain gap of HFR. Specifically, we first introduce a dual variational autoencoder to represent a joint distribution of paired heterogeneous images. Then, in order to ensure the identity consistency of the generated paired heterogeneous images, we impose a distribution alignment in the latent space and a pairwise identity preserving in the image space.